Annotation and classification of legal text are central components of empirical legal research. Traditionally, these tasks are often delegated to trained research assistants. Motivated by the advances in language modeling, empirical legal scholars are increasingly turning to prompting commercial models, hoping that it will alleviate the significant cost of human annotation. Despite growing use, our understanding of how to best utilize large language models for legal tasks remains limited. We conduct a comprehensive study of 260 legal text classification tasks, nearly all new to the machine learning community. Starting from GPT-4 as a baseline, we show that it has non-trivial but highly varied zero-shot accuracy, often exhibiting performance that may be insufficient for legal work. We then demonstrate that a lightly fine-tuned Llama 3 model vastly outperforms GPT-4 on almost all tasks, typically by double-digit percentage points. We find that larger models respond better to fine-tuning than smaller models. A few tens to hundreds of examples suffice to achieve high classification accuracy. Notably, we can fine-tune a single model on all 260 tasks simultaneously at a small loss in accuracy relative to having a separate model for each task. Our work points to a viable alternative to the predominant practice of prompting commercial models. For concrete legal tasks with some available labeled data, researchers are better off using a fine-tuned open-source model.

These and many other new insights are from Gartner Hype Cycle For AI, 2019 published earlier this year and summarised in the recent Gartner blog post, Top Trends on the Gartner Hype Cycle for Artificial Intelligence, 2019. Gartner's definition of Hype Cycles includes five phases of a technology's lifecycle and is explained here. Gartner's latest Hype Cycle for AI reflects the growing popularity of AutoML, intelligent applications, AI platform as a service or AI cloud services as enterprises ramp up their adoption of AI. Gartner advises its clients to consider including speech recognition on their short-term AI technology roadmaps. Gartner observes, unlike other technologies within the natural-language processing area, speech to text (and text to speech) is a stand-alone commodity where its modules can be plugged into a variety of natural-language workflows.

Gartner considers the following AI technologies to be on the rise and part of the Innovation Trigger phase of the AI Hype Cycle. AI Marketplaces, Reinforcement Learning, Decision Intelligence, AI Cloud Services, Data Labeling, and Annotation Services, and Knowledge Graphs are now showing signs of potential technology breakthroughs as evidence by early proof-of-concept stories. Technologies in the Innovation Trigger phase of the Hype Cycle often lack usable, scalable products with commercial viability not yet proven. Smart Robots and AutoML are at the peak of the Hype Cycle in 2019. In contrast to the rapid growth of industrial robotics systems that adopted by manufacturers due to the lack of workers, Smart Robots are defined by Gartner as having electromechanical form factors that work autonomously in the physical world, learning in short-term intervals from human-supervised training and demonstrations or by their supervised experiences including taking direction form human voices in a shop floor environment.